A Study of Cross - Validation and Bootstrapfor Accuracy Estimation and Model

We review accuracy estimation methods and compare the two most common methods: cross-validation and bootstrap. Recent experimental results on artiicial data and theoretical results in restricted settings have shown that for selecting a good classiier from a set of classi-ers (model selection), tenfold cross-validation may be better than the more expensive leave-one-out cross-validation. We report on a large-scale experiment|over half a million runs of C4.5 and a Naive-Bayes algorithm|to estimate the eeects of diierent parameters on these algorithms on real-world datasets. For cross-validation, we vary the number of folds and whether the folds are stratiied or not; for boot-strap, we vary the number of bootstrap samples. Our results indicate that for real-word datasets similar to ours, the best method to use for model selection is tenfold stratiied cross validation, even if computation power allows using more folds.